%
% $Id: sec3.tex,v 1.31.2.2 1996/06/13 21:57:15 carl Exp $
%
\newpage

\section{DATASET FILES AND SPECIFICATIONS}\label{sec-data}
\thispagestyle{plain}
\setcounter{figure}{0}
\chead[\fancyplain{}{\thesection.\ DATASET FILES AND SPECIFICATIONS}]
      {\fancyplain{}{\thesection.\ DATASET FILES AND SPECIFICATIONS}}

A dataset is a collection of \emph{cases}.  For each case, the values
of certain \emph{attributes} are recorded.  \delve{} stores these
attribute values in a file with a standard format that is general
enough that a wide variety of datasets can be represented without loss
of information.  For each dataset, \delve{} also keeps a specification
file, which records basic information such as the number of attributes
and their theoretical ranges.  Finally, the original files or programs
from which the dataset was derived are retained in the \delve{}
archive, along with any original documentation.

Files relating to a dataset are kept in a directory with the same name
as the dataset, located in the {\tt data} sub-directory of a top-level
{\tt delve} directory.  Some of the files that may appear in such a 
dataset directory are listed in Figure~\ref{fig-dataset-dir}.

\begin{figure}[b]

\rule{\textwidth}{0.5pt}

\hspace*{-4pt}\begin{tabular}{ll} \\[-6pt]
{\tt Summary} & A brief description of the dataset \\
{\tt Dataset.data} 
  & The actual data, in the format described in Section~\ref{data-format} \\
{\tt Dataset.spec}
  & Specifications for the dataset, usually accessed using the \dinfo\ command 
    \\[5pt]
{\tt Source} 
  & A sub-directory with files relating to the source of the dataset, such as:\\
\hspace{13pt}{\tt Notes}    
  & \hspace{13pt}Documentation on the dataset \\
\hspace{13pt}{\tt original}
  & \hspace{13pt}The original data file (but sometimes there will be more 
    than one) \\
\hspace{13pt}{\tt gen.c}    
  & \hspace{13pt}C program for generating dataset (or {\tt gen.f} for a Fortran
    program, etc.) \\[5pt]
$\!\!\!\left.\begin{array}{l} 
\mbox{\em Prototask-1} \\ 
\mbox{\em Prototask-2} \\ 
\mbox{\em Prototask-3} 
\end{array}\ \right\}$
  & Sub-directories for prototasks based on the dataset 
    (see Section~\ref{sec-task}) 
\end{tabular}

\caption{Some files and sub-directories that may appear within a \delve{}
         dataset directory.}

\label{fig-dataset-dir}
\end{figure}


\subsection{Specifications for datasets:~~The \dinfo{} command}\label{data-spec}

The specifications for a dataset include information about the dataset
as a whole, such as its origin and usage within \delve{}, plus
information about each attribute in the dataset, such as its range of
legal values.  This information is stored in the dataset's
specification file, \texttt{Dataset.spec}.  However, the only time you
will need to directly access this specification file is when you
create a new dataset, using the procedure described in
Section~\ref{data-prepare}.

Usually, it is more convenient to view the specifications for a
dataset using the \dinfo{} command, as was illustrated in the tutorial
in Section~\ref{intro-tutorial}.  For instance, to see the
specifications (as well as some other information) for the
\texttt{demo} dataset, you would use the command\vspace{-5pt}
\begin{Session}
   dinfo /demo 
\end{Session}\vspace{-5pt} 
Further details on individual attributes of the dataset can be
obtained by using the \texttt{-a} option with \dinfo{}, as is
illustrated in Figure~\ref{fig:dataset-dinfo-a}.

Note that dataset specifications contain only very basic information,
which is not likely to be wrong unless the data has been totally
misinterpreted.  More debatable prior information may be specified as
part of a task description (Section~\ref{task-prior}).

The following characteristics of a dataset as a whole are recorded
as part of its specification, and displayed by \dinfo{}:\vspace{-3pt}%
\begin{list}{}{%
\setlength{\leftmargin}{1.1in}%
\setlength{\labelwidth}{0.7in}%
\setlength{\labelsep}{0.1in}%
}
\item[{\tt Origin:}\hfill]
   {\tt natural} \OR {\tt cultivated} \OR {\tt simulated} \OR
   {\tt artificial}

A \emph{natural} dataset was originally gathered for some real-world
application; a \emph{cultivated} dataset comes from a real-world
source, but was never used to solve a real problem; a \emph{simulated}
dataset was generated by a simulator, but is believed to resemble real
data --- as opposed to an \emph{artificial} which is generated
according to some mathematical formula and does not pretend to
resemble any real dataset. These distinctions are discussed further in
Section~\ref{scope-data}.

\item[{\tt Usage:}\hfill]
  {\tt development} \OR {\tt assessment} \OR {\tt historical}
  \OR {\tt ?}

A \emph{development} dataset is recommended for use in developing new
learning methods, but to avoid bias, should not be used for formal
assessments.  An \emph{assessment} dataset is intended for use in
formal assessments; use for development should be minimized.  A
\emph{historical} dataset is included in \delve{} because it has been
used for assessing learning methods in the past, but is not
recommended for general use.  A `?' indicates that a recommended usage
has not yet been decided on.

\item[{\tt Order:}\hfill] 
  {\tt informative} \OR {\tt uninformative} \OR {\tt ?}

A dataset has an \emph{informative} ordering if the order of cases
may convey information that is not already present in the attribute
values.  The order is recorded as \emph{uninformative} if it is
random, or has some basis that is not related to any matter of
interest.  The order is recorded as `?' if the order appears to be
non-arbitrary, but the basis of the ordering cannot be determined from
the available documentation.

\item[{\tt Commonality indexes are present}] ~

If this line is displayed by \dinfo, {\em commonality indexes\/} are
associated with some or all cases in the dataset.  Cases with the same
commonality index share something in common, as is described further
in Section~\ref{data-dependencies}.  If this line is not displayed,
the cases in the dataset do not have commonality indexes.\vspace{-3pt}
\end{list}

If the ordering of a dataset is informative, or if commonality indexes
are present, the issue of possible dependencies between cases must be
addressed, as is discussed in Section~\ref{data-dependencies}.

\begin{figure}[t]
\begin{Session}
Dataset: /demo
Origin: artificial
Usage: development
Order: uninformative
Number of attributes: 5
Attributes: 
     #  name     c/u range        description
     1  SEX       u  male female  Sex of the person
     2  AGE       u  [0,Inf)      Age of the person in years
     3  SIBLINGS  u  0..Inf       Number of siblings the person has
     4  INCOME    u  [0,Inf)      The person's annual income (dollars)
     5  COLOUR    u  pink blue red green purple 
                                  The person's favourite colour
Prototasks: 
        age
        colour
        income
        sex
        siblings
\end{Session}\vspace{-4pt}
\caption{Output of the command: \texttt{dinfo -a /demo}.}
\label{fig:dataset-dinfo-a}
\end{figure}

Each dataset has a specified number of attributes associated with each
case.  Datasets in which the number of attributes varies from case to
case are not handled by \delve, though it is possible for the values
of some attributes to be missing in some cases (see
Section~\ref{data-format}).  The attributes for a dataset are numbered
from 1 on up.  Attributes can also have short {\em names}, which can
be used in place of numbers to identify them.  For the \texttt{demo}
dataset illustrated in Figure~\ref{fig:dataset-dinfo-a}, the attributes
have names of \texttt{SEX}, \texttt{AGE}, etc.

The dataset specification also records whether each attribute was
\emph{controlled} or \emph{uncontrolled} (abbreviated to `{\tt c}'
or `{\tt u}' in the output of \dinfo).  The values of a controlled
attribute were fixed for each case by the investigator who gathered
the data; the values of an uncontrolled attribute were not fixed,
though the investigator will often have had some influence on the
mechanism by which they were generated.  For example, in a dataset
concerning the growth of plants under various conditions, the amount
of fertilizer applied to a plant would usually be a controlled
attribute, whereas the amount of rainfall would be an uncontrolled
attribute.  This field will be recorded as `?' if it is not clear from
the available documentation whether or not the attribute was
controlled.

Each attribute in the dataset also has a specified \emph{range},
consisting of a list of items, each of which defines a set of allowed
values for the attribute.  Such an item can specify a single permitted
value (which could be a \emph{missing value}, as discussed in
Section~\ref{data-format}), or a set of permitted numerical values
having the form of an open, closed, or half-open interval of real
numbers, or a range of integers.  The bounds of a real interval can be
ordinary numbers, or one of `{\tt Inf}', `{\tt -Inf}', or `{\tt
+Inf}', with `{\tt Inf}' representing infinity; these bounds are
enclosed by round or square brackets, indicating whether the bound
itself is included.  For example, {\tt [0,1)} represents the interval
from 0 to 1, including 0, but not including 1, and {\tt (0,Inf)}
represents the set of positive real numbers.  An integer range
extending from \emph{low} to \emph{high}, inclusive, is written as
\emph{low{\tt ..}high} (with no enclosing brackets); \emph{low} and
\emph{high} can be infinite, as for real intervals.  For example, {\tt
1..Inf} represents the positive integers.

Several items can be combined, as in the following range:\vspace{-5pt}
\begin{Session}
(-Inf,0)  (0,+Inf)  ?
\end{Session}\vspace{-5pt}
This specifies that the attribute can take on any numerical value
other than zero, as well as the missing value indicator,
`\texttt{?}'.

Note that the range specified for an attribute is the full set of
conceivable values, regardless of whether all of these values actually
occur.  For example, the range {\tt [0,100]} would be appropriate for
an attribute that represents the percent by weight of water in a
sample of some substance, since it is inconceivable that the value
could ever fall outside this range, but any more narrow range would
not be appropriate, even if the actual values in the dataset 
never exceed 10\%.  Similarly, for an attribute representing a person's
birth sign, the appropriate range would be all twelve signs of the
zodiac, even if no Scorpios happen to be included in the dataset.

Finally, an attribute may be accompanied by a short \emph{description},
which is ignored by the \delve{} software, but may help users keep track
of which attribute is which.


\subsection{Datasets with dependencies between cases}
\label{data-dependencies}

Dependencies between cases in a dataset are of significance for two
reasons.  First, a learning method may take account of such
dependencies in order to improve learning.  For example, a method that
adapts its behaviour based on the size of the training set might
consider the effective size of the training set to be reduced when
training cases are dependent (since the information in one case may
largely duplicate the information in other cases).  Second, \delve{}
itself must be aware of possible dependencies in order to avoid
assessing learning methods using test cases that are dependent on the
cases included in the training set, and in order to properly compute
standard errors for performance figures.

Whenever a dataset has an informative ordering, there is the
possibility of \emph{sequential dependencies} between the cases.  In
some circumstances, however, this possibility may be remote enough
that it is reasonable to ignore it --- for example, if the cases are
ordered by the time when their attributes were measured by some
machine, it is possible that dependencies are present as a result of
temporal variation in the machine's accuracy, but this possibility
may be too remote to be worth worrying about.

Dependencies between cases may also exist whenever \emph{commonality
indexes} are present.  Cases with the same commonality index have
something in common of a nature that may produce dependencies.  For
example, suppose the problem is to classify cars by make, given an
image of the car.  If several cases were obtained by viewing the
\emph{same} car from different angles, the whole group of cases should
be used either for training or for testing, but not for a mixture of
these.  Otherwise, a test case might be correctly classified based on
some idiosyncratic feature of a training case in the same group (eg, a
scratch on the car's bumper).  Similarly, in a dataset of spoken
words, all the words spoken by one person would share a commonality
index.

The presence of commonality indexes or of an informative ordering is
merely an indication of the possibility of dependencies, and even if
dependencies exist, they may or may not be of significance in the
context of a particular learning task.  More specific information
concerning dependencies may be given in prototask and task
specifications.  When significant dependencies do exist, they are
dealt with in \delve{} in one of two ways.  One is to properly
accommodate the dependencies, as would be necessary in a real-world
learning task.  The other is to randomly select cases so as to produce
an internally-consistent task without dependencies.  Such tasks can be
useful for assessing learning methods even though they no longer
correspond to a real-world situations.  These issues are discussed
further in Section~\ref{sec-task}.

{\em Note: Currently, commonality indexes are not really implemented ---
you can include them in \delve{} dataset files, but they will be ignored.
Also, the only way of dealing with sequential dependencies at present 
is to randomize the ordering.}


\subsection{The \delve{} format for dataset files}\label{data-format}

DELVE datasets are stored in a standard format that is designed to
preserve as much relevant information from the original data as
possible, even if some of this information is not currently used by
DELVE.  Users may occasionally wish to look at these dataset files,
but programs implementing learning methods do not read these files
directly.  Instead, a learning method will work with data files that
have been appropriately encoded for a given task, as described in
Section~\ref{sec-assess}.

A dataset in the \delve{} standard format consists of an ordered list of
\emph{cases}, each of which consists of values for an ordered list of
\emph{attributes}.  A case may optionally be accompanied by a
\emph{comment}, which may be anything, and by a \emph{commonality
index}, a number that identifies several cases as having a common
origin.  \emph{Note: Commonality indexes aren't implemented\nolinebreak{} yet.}

The number of attributes is a characteristic of the dataset, and all cases 
have values (of some sort) for all attributes.  The value of an attribute
may be any of the following:\vspace{-5pt}
\begin{itemize} 
\item A string that represents a number in any of the common forms
      --- that is, with syntax
      \[
         [\ \mbox{\tt +}\ |\ \mbox{\tt -} \ ]\ 
         [\ digit\ldots\ ]\ [\ {\tt .}\ [\ digit\ldots\ ]\ ]\
         [\ (\ \mbox{\tt e}\ |\ \mbox{\tt E}\ )\ 
         [\ \mbox{\tt +}\ |\ \mbox{\tt -}\ ]\ digit\ldots\ ]
      \]
      with the restriction that at least one digit must appear, not 
      counting digits after an `{\tt e}' or `{\tt E}'.
\item A number as above, preceded or followed by `{\tt :}',
      representing a \emph{censored value}.  If the colon is at the end,
      the actual value of the attribute is known only to be greater than
      or equal to the given number; if the colon is at the beginning, the
      actual value is less than or equal to the given value. \emph{Note:
      Support of censored values is not yet implemented.}
\item The character `?', perhaps followed by other non-space characters.
      This represents a \emph{missing value}.
      The other characters may indicate the reason for the value being 
      missing.  Just `?' is used for values that are missing due to
      a random mechanism unrelated to the relationship of inputs to targets. 
      \emph{Note: Missing values are not really implemented yet. About the only
      thing useful that can be done at present with cases having missing 
      values is to ignore them.}
\item Any other string of non-space characters that does not begin with
      `{\tt $\backslash$}', `{\tt @}', `{\tt \#}', `{\tt (}', 
      `{\tt [}', `{\tt +}', `{\tt -}', `{\tt .}', `{\tt :}', 
      or a digit. These strings represent values from a discrete set of 
      categories.
\end{itemize}\vspace{-5pt}
Numerical values are represented in as close to their original form as
possible --- for example, `5.0' is \emph{not} converted to `5' or to
`5.00'.  This preserves any information that might be contained in the
original choice of the number of significant digits.

A dataset in standard format is encoded as a ASCII file, in which the
cases appear in order, with each case being represented by a group of
lines.  All lines in a group except the last end with a space followed
by the character `{\tt $\backslash$}'.  The whole group of lines for a
single case should be thought of in terms of the single line that
would result if the `{\tt $\backslash$}' and the following newline were
removed.  Within the line (or group of lines) representing a case, the
attribute values appear in order, separated by one or more spaces.

If a case has a commonality index associated with it, it appears after
all the attributes.  This index consists of the character `@'
followed by one or more digits.  

If a case has a comment associated with it, it appears at the end of
the line, preceded by `{\tt \#}'.  These comments are ignored by 
all \delve{} programs.


\subsection{Preparing a new dataset:~~The \dcheck{} command}\label{data-prepare}

When a dataset is obtained, the original data files, documentation,
programs, and any other possibly relevant material should be saved in
as close to its original form as possible.  This archived information
may be of interest if, for example, doubts should arise as to whether
the original data format was properly interpreted, or questions are
raised regarding the real-world relevance of the data.  This information
goes in the {\tt Source}\ sub-directory of the dataset's directory.

The dataset should then be converted to the standard \delve{} format,
and stored in the {\tt Dataset.data} file in the dataset's directory.
The aim in doing this should be to retain all information that could
be relevant to some use of the data, discarding only fields such as
redundant case numbers.  Converting a dataset will often be simply a
matter of mechanically reformatting it.  However, difficulties of
interpretation may arise if there are peculiar aspects to the original
data, or if it is inadequately documented.  In such cases, the
rationale for the decisions made should be documented, in the {\tt
Notes} file in the {\tt Source} directory for the dataset.

As well as the data file itself, you must create a specification file
for the dataset, with the name \texttt{Dataset.spec}, which describes how the
dataset is to be interpreted and used.  The specification file is
meant to be machine readable, and, as such, has a very strict format.
The file may have zero or more initial comment lines (lines where the
first character is a \verb+#+).  Immediately after the comments lines
there should appear the three lines (in any order):\vspace{-5pt}
\begin{Session}
Origin: {\rm \em origin}
Usage:  {\rm \em usage}
Order:  {\rm \em order}
\end{Session}\vspace{-5pt}
These lines specify the information discussed in Section~\ref{data-spec}.
Specifically:\vspace{-5pt}
\begin{list}{}{%
\setlength{\leftmargin}{0.9in}%
\setlength{\labelwidth}{0.5in}%
\setlength{\labelsep}{0.18in}}
\item[{\em origin\hfill}]
    should be one of the strings \texttt{natural},
    \texttt{cultivated}, \texttt{simulated}, or 
    \texttt{artificial}.\vspace{-2pt}
\item[{\em usage\hfill}]
    should be one of the strings \texttt{development},
    \texttt{assessment}, \texttt{historical}, or \texttt{?}.\vspace{-2pt}
\item[{\em order\hfill}]
    should be one of the strings \texttt{informative},
    \texttt{uninformative}, or \texttt{?}.\vspace{-5pt}
\end{list}\vspace{-5pt}
In addition to the above lines, you may include the optional line:
\begin{Session}
Title: {\rm \em title}
\end{Session}\vspace{-5pt}
where \textit{title} is a string describing the dataset.  It is not
used directly by \delve{}, but it is available to users via \texttt{dinfo}.

The string \texttt{Commonality indexes are present} may appear on the next
line.  If there are no commonality indexes, this line should be omitted.
{\em Note: Currently, this line must be omitted.  You can always include
commonality indexes, but they will be ignored.}

Following these lines should be a line contain the single string
\texttt{Attributes:}.  Each remaining line in the file will be
interpreted as an attribute description, with the format:
\[
\mbox{{\em i name control range} [ {\tt \#} {\em comment} ]}
\]
The fields above have the following meanings:\vspace{-3pt}
\begin{list}{}{%
\setlength{\leftmargin}{0.9in}%
\setlength{\labelwidth}{0.5in}%
\setlength{\labelsep}{0.18in}}
\item[{\em i\hfill}] is the integer index for the
    attribute. Indices should start at one and increment by one for
    each line.
\item[{\em name\hfill}] is a mnemonic name that can be used in place
    of the attribute's index.  The names must be unique (within a
    dataset).  They may not contain spaces, and may not look like 
    integers.
\item[{\em control\hfill}] is one of the characters \texttt{c} or
    \texttt{u}, depending on whether the attributes was controlled or
    uncontrolled.
\item[{\em range\hfill}] is the range for the attribute, a list of items of
    the form described in Section~\ref{data-spec}.
\end{list}\vspace{-3pt}
The range for an attribute may optionally be followed by `{\tt \#}'
and a comment describing the attribute.

The specification file for the \texttt{demo} dataset is shown in
Figure~\ref{fig:dataset-spec}.

\begin{figure}[t]
\begin{Session}
Origin: artificial
Usage: development
Order: uninformative
Attributes:
 1 SEX      u male female  # Sex of the person
 2 AGE      u [0,Inf)      # Age of the person in years
 3 SIBLINGS u 0..Inf       # Number of siblings the person has
 4 INCOME   u [0,Inf)      # The person's annual income (dollars)
 5 COLOUR   u pink blue red green purple  # The person's favourite colour
\end{Session}
\caption{Dataset specification file for the \texttt{demo} dataset.}
\label{fig:dataset-spec}
\end{figure}

Once you have created both \texttt{Dataset.data} and
\texttt{Dataset.spec}, you should check that the two are legal and
consistent using the \dcheck{} command, which will verify that each
case has the right number of attributes, and that they are in the
specified ranges. Note that missing values are allowed in
\texttt{Dataset.data} only if they are listed as allowed in
\texttt{Dataset.spec}.  A censored value for an attribute (specified
using `:') is allowed only if it includes at least one possible
value that is within the attribute's range.
\emph{Note: The \dcheck{} command is not implemented yet.}
